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Abstract 

The dynamical organization in the presence of noise of a Boolean 
neural network with random connections is analyzed. For low levels of 
noise, the system reaches a stationary state in which the majority of its 
elements acquire the same value. It is shown that, under very general 
conditions, there exists a critical value ij c of the noise, below which the 
network remains organized and above which it behaves randomly. The 
existence and nature of the phase transition are computed analytically, 
showing that the critical exponent is 1/2. The dependence of r\ c on 
the parameters of the network is obtained. These results are then 
compared with two numerical realizations of the network. 

Key words: Neural Network, Phase Transition, Noise, Organization. 
Running title: Dynamical phase transition in a neural network with noise. 



* Corresponding author, e-mail: maximino@control.uchicago.edu; tel: (773)-702-0946; 
fax: (773)-702-5863 



1 



1 Introduction 



Boolean networks have been used to describe a wide variety of complex sys- 
tems. They provide a common language for models of associative memory 



IS, O, ffQ], spin glasses [17, 0, O, pfl, dynamics of evolution [21, 12, 11], and 



cellular automata || j|, [L3|, |22|. A typical Boolean network model consists of 
a set of binary elements (also called nodes, neurons or spins, depending on 
the context) which are connected among them to form a net. The value of 
each element at a given time depends on the value at the previous time step 
of all the nodes that are connected to it. 

The use of common tools of statistical physics has revealed a strong paral- 
lel between Boolean networks and dynamical systems. In this context, several 
authors have studied the non-equilibrium dynamics of deterministic Boolean 
networks and, in particular, the one of neural networks 0, [14|]. Their work 
has unveiled the existence of a variety of possible collective behaviors such as 
synchronized oscillations or chaos [ffl, |2(], [It]]. On a similar perspective, the 
influence of noise on the dynamics of Boolean networks has been analyzed 
in ]TB| , |9|. It is shown that, for increasing noise level, the barriers separating 
different attractors decrease and then disappear. 

We are interested in considering the changes in the dynamical properties 
of a deterministic system in the presence of noise. Following this motivation, 
we study a simple Boolean network model exhibiting self-organization and 
analyze its tolerance to the effect of noise. We show analytically that the 
system undergoes a dynamical second-order phase transition as its amount 
of randomness is increased. 

The paper is organized as follows. In section |2|, we describe the neural 
network model with noise. Section [3] introduces the definition of the order 
parameter characterizing the network and presents numerical evidence for 
the phase transition by considering two particular network examples. In 
section |4] we find the dynamical equation satisfied by the order parameter 
and compute analytically its fixed points, exhibiting the phase transition. In 
section ^| we apply these results to the two cases studied in section |. Finally, 
section || is our conclusion. 
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Figure 1: Schematic representation of the structure of the network. Every element 
crj is connected to K other elements {a^ , . . . , Gi K } (the linkages), which are chosen 
at random from the entire set {&i}i=i...N ■ The contribution of each <7j. element to 
the input function of is weighted by Cij as given in equation ([[]) . 



2 Definition of the model 

Consider a neural network composed of N elements {oq, cr 2 , . . . , & N }, each of 
which can only take the values <7j = —1 or Oi — +1. Every (Tj is randomly 
connected to any elements of the network, which define its set of linkages 
{&% }j=i,...,K ( see T ne parameter is the connectivity of the network. 

Each linkage ov is weighted by an independent random variable that 
is chosen with a probability density function (PDF) given by P c (x). The 
NK connections of a network, and its corresponding weights, remain fixed 
throughout the evolution of the system. 

At every discrete time step, each <7j receives a signal +1 or —1 equal to 
the input function 

f (ca, . . . , cue', o- h ,..., o iK ) = Sign I r, ; rr, J . (1) 

For the particular case in which we have for all weights Qj = 1, this definition 
corresponds to the majority rule, in which / takes the same value as the 
majority of the linkages. 

Using the input function ([!]), we define a stochastic evolution rule for 
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every cxj by introducing a noise intensity r\ such that 



/ (qi . . . c iK ; a, h (t) ... (J iK (t) ) with probability 1 - 77 



<7i(t + l) 



/ (c a . . . c iX ; cTii (*)••■ Oi K (t) ) with probability rj. 



(2) 



The dynamics can thus be set from purely deterministic to purely random by 
varying rj between and 1/2. Note that in the case with r\ = 0, the system 
will typically converge to an ordered state in which all the <7j are equal. 

Due to the presence of noise and to the randomness in the initial as- 
signment of the linkages, the statistical properties of the dynamics of the 
network do not change if the connection weights or the linkages are either 
time-independent or if they are randomly re-assigned at every time step. Us- 
ing the language of boolean networks, this means that for the model presented 
here the annealed and quenched dynamics are equivalent 0. 

3 Numerical evidence 

In this section we perform a numerical study of the evolution of the neural 
network model introduced above. We show that the system undergoes a 
dynamical phase transition (for iV — > oo) from an ordered to a disordered 
state as the noise intensity r\ is increased. The analytical expression for this 
transition will be deduced in section f|. 

Let us define an order parameter that adequately describes the degree of 
alignment of the elements of the network. We first introduce 



With this definition, \s(t)\ ~ 1 for an "ordered" system in which most el- 
ements take the same value, while \s(t)\ ~ for a "disordered" system in 
which the elements randomly take values +1 or —1. For systems where the 
time-average of \s(t) \ converges, a time-independent order parameter can now 
be defined as 




1=1 



(3) 





where T can take any arbitrary finite value without changing ty. 
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Figure 2: Bifurcation diagram of the order parameter $ as a function of the noise 
intensity r/ for a neural network model in which Cy = 1 for all weights. A phase 
transition occurs at ij c ~ 0.3153. The numeric and analytic results only differ at 
7) ~ rj c , where the blowup shows a slight difference due to finite size effects. 

We computed numerically the evolution of the model for a network with 
N = 100000 elements, connectivity K — 11, and random initial conditions. 
In practice, the order parameter \l/ was obtained by integrating \s(t)\ from 
T = 1000 (to drop the initial relaxation dynamics) until T = 10000. A 
change to a larger integration time produces negligible variations on the 
result. 

The numerical results presented on Figure [| show the bifurcation diagram 
of ^ as a function of the control parameter r] for the case with c^- = 1, in 
which all connection weights are equal. The input function then simply 
becomes the majority rule. It is apparent that the system undergoes a phase 
transition at r] c « 0.32. For rj < rj c , all elements in the system will tend 
to align either to +1 or to — 1. For rj > rj c , their values are randomly 
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Figure 3: Bifurcation diagram of the order parameter $ as a function of the 
noise intensity r\ for a neural network model where the weights Cij follow the PDF 
defined in (|5|). A phase transition occurs at ij c ~ 0.2838. The blowup shows again 
the differences between the numeric and analytic results due to finite size effects 
(see Fig. |). 

distributed. The blowup on figure || shows the usual finite size effect on the 
phase transition, which smoothes the curve near r\ c . 

On figure [| we present the results for the case of a network with fixed 
connection weights that follow the PDF 

P(x) = l 1 if °^ x ^ 1 ( 5 ) 
I otherwise. 

The phase transition on this system is qualitatively equivalent to the previous 
one, but the critical noise value is now changed to r\ c « 0.28. The blowup 
shows again the finite size effects. 
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4 Analytic solution 



We will compute here the exact analytic expression that relates the noise 
intensity control parameter r\ and the order parameter \l/ for any given PDF 
of the connection weights P c (x). 



4.1 Dynamics of the order parameter 

First, we will relate the probability distribution of the system at a time t + 1 
with the one at a time t. In order to do so, let us define 07v(t) as the fraction 
of elements in the network whose value is +1 at time t: 

i=i 

In the thermodynamic limit iV — > oo, the above quantity transforms into the 
probability that at time t any arbitrary node <Ji acquires the value +1: 

( f ) (t)=P t {a i = +l}= lim <f> N (t) (7) 

Note that from definition (0), the relation between s(t) and </>(t) is simply 
given by 

s(t) = 2<f)(t) - 1. (8) 

Thus, in a fully ordered state we have \s(t)\ = 1 and <p(t) = or 1, while in 
a fully disordered state we have \s(t)\ = and <f>(t) = 1/2. 

It is useful to define as the argument of the Sign function appearing 
in the definition of / for the z-th element at a time t (see equation (JT|) ) , 

K 

Ci(*) = E^(*)- ( 9 ) 

3=1 

If the linkages of every node are assigned in a sufficiently random wayQ, 
the products CijaiAt) can be considered as independent random variables. 
Therefore, if we denote by P^(x) and P CCT (t)(^) the PDF associated to & 

1 If the linkages are not assigned randomly, the situation changes. For example if they 
are chosen among the first neighbors of every element, there is no phase transition and 
the analysis presented here is not applicable. 
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and to the product cyc^ respectively, then P^ t )(x) is simply given by the 
X-fold convolution of P C a{t) {%) with itself: 

P ?(t)(» = P c<7(t) * P«r(t) * • • • * P«r(t)(aO • ( 10 ) 

a: times 

In terms of P^ t )(x), the probability J(t) of having the input function / = +1 
at time t can be computed as 

J(*) = P t {/ = +l}= / P« t )(a;)tte. (11) 

" 

Using the updating rule (0), we can now directly write the probability 1) 
of having <Ji(t + 1) = +1 in terms of I(t) and r\: 

0(t + l) = I(t)[l- V } + [1- 1(f)] 17. (12) 

This master equation describes the stochastic dynamics of the network. Its 
fixed points as a function of rj will generate the bifurcation diagram showing 
the phase transition. 



4.2 Probability distribution of the input function 



In order to express equation (O) in a closed form, we must find how I(t) 
relates to (pit). For this, we first compute P^m(x) which is in turn determined 
by P ccr ( t )(x). In Fourier space, the convolution appearing in ( TO ) acquires the 
simple form 



(A) 



co-(t) 1 



K 



(13) 



where Pm) (A) and P CCT (t) (A) are the Fourier transforms of P^ t ) (x) and P 



co-(t) 



x 



respectively. 

Since the connection weights c^- are distributed according to the proba- 
bility function P c (x), and the variables <Ji{t) evaluate to +1 with probability 
(p(t) and to —1 with probability 1 — <p{t), it follows that the PDF of the 
products CijCTj. is given by 



Pc( t )(z) = <f>(t)P c (x) + [1 - <P{t)} P c (-X) 



(14) 
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Taking the Fourier transform of the previous expression, and inserting the 
result into equation (]3~3|) , one gets 



(A)= p:(a) + (p c (a) - p:(a)) m 



K 



(15) 



where the '*' denotes complex conjugation. From the above expression it 
is apparent that P^m(x), and consequently I(i), are polynomial functions 
of 4>(t). Therefore, equation (12) is a polynomial of degree K in <p(t), with 
solutions that depend on the value of the noise intensity rj. As we will see 
later, the roots of this polynomial will furnish the bifurcation diagram of the 
order parameter. 

For convenience, we will write our results in terms of s(t) instead of 4>(t) 
(see equation (||)). Substituting 0(t) = [s(t) + 1] /2 in expression ( |15|) we get 



(A) 



P C (A) + P:(A) , P C (A)-P C *(A) 



s(t) 



K 



(16) 



By denoting g(X) and h(X), the real and imaginary parts of P C (A) respectively, 
p£(t)(A) can then be written as 



P«t)(A) = [g(X)+ih(X)s(t) 

K f K \ 



K 



[g(X)} K - m th(X)s(t) 



m=0 



171, 



whose inverse Fourier transform is 

K 



^ ' 211(711, 



m=0 



[g(X)f- m h(X) 



iXx 



dx \[s(t)} m . (17) 



We are now in position of computing I(t). Using equations (|TT| ) and (P^|), 
and moving the integrals inside the sum we have 



K 



I(t) =Y,a m [s(t)r 

m=0 



(18) 



where the a m are constant coefficients that depend only on P C (A) and are 
given by 



2n\m, 



K—m 



h(X)] m e~ iXx dXdx. 



(19) 
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Figure 4: Integration contour C on the complex plane that was used to compute 
(P). The pole at the origin is circumvented by the half-circle C\ of radius r. 



-i\x 



We can formally integrate over x by replacing e" 
the new x-integral, and then evaluating the result at e 
for the a m coefficients gives 



-(i\+e)x 



computing 



0. Our final formula 




[g(X)] K ~ m h(X) 



dX. 



(20) 



Note that a m = for all even values of m. Indeed, since the function <?(A) is 
even and h(X) is odd, the integrand will be antisymmetric for any even m, 
thus vanishing the integral. 

It will be useful for later calculations to compute the value of a . In fact, 
it turns out that a = 1/2 for every K and any "well behaved" function 
P c (x). This can be readily proven by extending the integral in ( p0|) to the 
complex plane. The integration path C shown on figure [| contains no poles, 
therefore 



K 



dz = 0. 



(21) 



Since P c (x) is a PDF, Parseval's theorem guarantees that its Fourier trans- 
form is square integrable, which implies Pf^ g 2 (X)dX < oo. Therefore, the 
contribution of segment C3 to the integral in ( f2T| ) is zero for R — > 00. On 
the other hand, being (7(A) the real part of the Fourier transform of a PDF, 
one has ^(0) = 1 and thus, for any well behaved function it is possible to 
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approximate g{r) ~ 1 over the small segment C\. This allows us to compute 
f c - [g(z)\ dz = J®id9 = —in. Replacing into (|2l|) , we obtain the value of 
the integral in ( |2Ti| ) and find that a = 1/2 for any P c (x) and any K. 

4.3 Computing the bifurcation diagram 

We can now combine the main results of sections |4.1| and [4 . 2| to find an 
analytic expression relating rj and We have shown that I(t) is a polynomial 
of degree K in s(t). Therefore, in terms of s(t), the master equation (|12]) 
governing the dynamics of the network becomes 

s(t + 1) = 2 (1 - 2 V ) (a lS (t) + a 3 [s(t)} 3 + ■■■ + a K [s(t)} K ) . (22) 

In the limit t — > oo, s(t) will asymptotically approach a fixed point s which, 
from the above equation, obeys 

s = 2 (1 - 2tj) (ais + a 3 s 3 + ■■■ + a K s K ) . (23) 

It is important to point out that if the probability function P c (x) is symmet- 
ric, there is no phase transition. Indeed, if P c (x) satisfies P c (x) = P c (— x), 
then the imaginary part h(X) of its Fourier transform would be identically 
zero. It follows from equation ( |2T)| ) that a m = 0Vm and, therefore, equations 
( |22|) and ( p3|) give the trivial result s = as the only possible solution for 
the dynamics. 

Equation (|23|) is always satisfied by s = 0. However, as i] is varied, this 
solution becomes unstable as other ones appear. Discarding the solution 
s = and solving equation ( p3| ) for r/, we get 

= ai - 1/2 + a 3 s 2 + ■■■ + a K s K - 1 
V ~ 2(a 1 + a 3 s 2 + --- + a K s K - 1 ) ' 1 J 

By dividing the polynomials and neglecting the terms of order s 4 and higher 
in the resulting expression^], we obtain 

M = lf 2 (25) 

2 As it can be seen in figures ^| and the order parameter vanishes continuously when 
passing from the ordered to the disordered phase. Therefore, in the vicinity of the phase 
transition s«0. 



11 



where r] c is defined as 

Since 03 < 0, equation (p5|) implies that real non-zero solutions for s in (^3]) 
only exist if rj < r] c . For rj > r] c the solutions of equation fl25|) are imaginary 
and therefore s = is the only acceptable solution of (|23|) . We thus conclude 
that a phase transition with critical exponent 1/2 will occur at rj = i] c . The 
explicit behavior of the order parameter \P = |s| near the transition will be 

(Vc - v) 1/2 for v < Vc 

(27) 

for 77 > rj c 

5 Examples of the analytic solution 

In this section we compare our analytic solution with the numerical results 
presented in section |3|. 

We consider first the case with Cjj = 1 for all connections. The PDF of 
these connection weights is then P c (x) = 5(x — 1), and we have 

rod 

P C (A) = / 6{x- l)e iXx dx = cos(A) + i sin(A). (28) 
Therefore, the coefficients a m are given by 



;m+l 



K\ r°° 1 



2n \m, 



f°° \ [cos(A)]^- m [sin(A)] m dX. (29) 
J— 00 A 



For the other case, in which the connection weights are uniformly distributed 
in the interval [0, 1] as given in expression (|5|), we get 

P C (A) = f 1 e^dx = + /- C " S(A) , (30) 

Jo A A 

and the corresponding coefficients a m are given by 
-i m+l (K\ r°° 1 



2n \m , 



T^TT [sin(A)] X " m [1 - cos(A)] m dX. (31) 
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Figure 5: Log- log plot of the order parameter $ as a function of the distance 
to the critical noise value rj c — rj. The left side graph corresponds to the case 
with equal connection weights Cjj = 1. The right hand graph is for uniformly 
distributed connection weights. The values of rj c = 0.3153 (left) and rj c = 0.2838 
(right) were obtained from equation (|26|). The numerical results (circles) and the 
analytic solution (solid line) approach asymptotically the dashed line representing 
the bifurcation form (f27|). 



We evaluated the integrals in ( |29| ) and fl3~T|) for the case K = 11, which 
was studied in section |3|. By replacing them into equation (24) we explicitly 
obtain 77 as a function of s for these two particular cases. The solid curves on 
figures |2| and |3] show the analytic bifurcation diagram that is found through 
this procedure. The agreement with the results our numerical simulation is 
excellent, confirming our assumption that all elements can be considered as 
independent random variables, even if the network connections and weights 
are kept fixed throughout the evolution of the system. 

Once the values of the coefficients obtained, the critical noise value 

T] c can be readily calculated by using equation ([26]). Figure |5] shows a log- log 
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Figure 6: Critical noise level r/ c for various values of the connectivity K and two 
different weight distributions: Cy = 1 for all connections (circles) and Cj,- uniformly 
distributed in [0, 1] (squares). As the connectivity increases, the amount of noise 
needed to disorganize the system must increase. For large K, the value of rj c will 
tend asymptotically to the maximum noise level 1/2. For K < 2 both cases have 
rj c = 0, and the system becomes disorganized for any noise r] > 0. In the constant 
weight case, the values of r\ c at consecutive odd and even values of K are equal 
(see text). 

graph of the vanishing value of ^ as a function of f] c — V> both for the constant 
connection weight case (j] c ~ 0.3153) and for the uniform distribution weight 
case (j] c ~ 0.2838). These results are the same that were presented in section 
|3|. On the displayed windows, finite-size effects around r\ c are not visible. As 
can be seen on this figure, the numerical results coincide perfectly with the 
analytic solution (|24j) (solid curve) and with the asymptotic behavior given 
in ( p7|) (dashed line). 

Figure 10 shows the critical noise level Tj c clS db function of the connectivity 
K for the two particular cases studied. As the connectivity is increased, the 
phase transition appears at higher levels of noise and it is increasingly hard 
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to disorder the system. As the system becomes more and more correlated, 
r] c will asymptotically approach its maximum value of 1/2 (data not shown). 
For the case in which Cy = 1 for all connections, the majority rule is not well 
defined for even values of K, since there can be equal number of elements 
with Uj = +1 and <7j = —1. This justifies the degeneracy observed for t] c with 
respect to consecutive odd and even values of K. For K = 2 the system will 
be disorganized at any r\ c > and no phase transition occurs. 



6 Conclusions 

We have shown the existence of a noise-driven phase transition in a neu- 
ral network with random connections. We found an exact analytic solution 
of the stochastic equation which governs the dynamics of the system. By 
finding its fixed points as a function of noise we constructed the bifurcation 
diagram, which shows that the phase transition is of second-order with a 
critical exponent of 1/2. 

Besides the randomness of the network connection, our work was carried 
out with very few assumptions. One of these was to consider the connection 
weights as statistically independent variables. This condition, however, is not 
satisfied by many models of interest such as the Hopfield neural networks [[It] 



It would therefore be of great interest to generalize our approach to cases in 
which the connection weights are statistically correlated. Given that these 
conditions are satisfied, the existence of the phase transition only seems to 
require that the PDF of the connection weights P c (x) is a non-symmetric but 
otherwise arbitrary function, as shown on section [4.3| . 

The general framework under which the results on this paper were de- 
rived leads us to believe that this type of noise-driven phase transitions from 
an ordered to a disordered state must be a robust feature of systems with 
elements that are somehow randomly connected M. 
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